(C) 2017 by Damir Cavar
This is a tutorial related to the discussion of feature extraction for classification and clustering in the textbook Machine Learning: The Art and Science of Algorithms that Make Sense of Data by Peter Flach.
This tutorial was developed as part of my course material for the course Machine Learning for Computational Linguistics in the Computational Linguistics Program of the Department of Linguistics at Indiana University.
In [1]:
from nltk import word_tokenize
In [25]:
text1 = """The city will pay for it by taxes on properties selling for more
than $5 million.
The real estate transfer tax, as it's called, was increased last year for both
residential and commercial properties. The hike was approved by voters in
November.
Powered by SmartAsset.com
SmartAsset.com
The tax starts at 2.25% and goes up to 3% for properties worth at least $25
million. It's expected to bring in an average of $45 million a year, according
to the city controller. But the money goes into the city's general fund and is
also expected to be used for affordable housing and senior support services.
The free tuition plan is expected to impact about 28,000 residents who currently
take classes at City College of San Francisco and encourage more people to sign
up. Chancellor Susan Lamb said the school has the capacity for 85,000 students.
It's difficult to predict how many more people will enroll, and how much the
free-tuition plan will end up costing. San Francisco has committed $5.4 million
a year for the next two years, and then will have to reassess. That includes a
one-time $500,000 stipend to City College to help handle an influx of students.
Related: Why New York's 'tuition-free colleges' still cost $14K
San Francisco's tuition-free plan is more progressive than others round the
country. First, everyone is eligible as long as they have resided in San Francisco
for at least one year.
It covers the $46 cost per credit no matter how rich you are, "even to the
children of the founders of Facebook," said city lawmaker Jane Kim.
You don't have to be enrolled full-time or be a recent high school graduate.
This means that people who are seeking job retraining or want to take a few
foreign language courses won't have to pay for the cost of the credits.
Related: Rhode Island governor wants to make college free, too
Students will still be on the hook for the mandatory $17 per semester fee at
City College and the cost of books, so college won't necessarily be free.
What also sets apart San Francisco's plan is that it offers the poorest students
additional money to help pay for these other expenses. An individual has to earn
less than $17,000 a year to qualify for the aid, or less than $37,000 for a
family of four. Eligible full-time students will get $500 a year and part-time
students will get $200 a year.
"We have the fastest growing income gap than any city across the nation," Kim said
on Monday at a press conference.
"Making city college free is going to provide greater opportunities for more San
Franciscans to enter into the middle class and more San Franciscans to stay in the
middle class if they currently are," she said.
The push for free tuition is gaining support across the country. Tennessee started
offering free community college to residents in 2015, and will expand the program
this year to include adults returning to school. Lawmakers in New York are
discussing a program that would make four-year and two-year public colleges
tuition-free for residents who earn less than $125,000 a year. And Rhode Island's
governor is pushing for two free years at public colleges for recent high school
graduates."""
In [66]:
tokens1 = word_tokenize(text1.lower())
In [67]:
from collections import Counter
In [68]:
fp = Counter(tokens1)
print(fp)
In [69]:
model = [ (i, fp[i], len(i)) for i in fp ]
print(model)
In [70]:
for x in model:
print( "\t".join( (str(x[1]), str(x[2]), x[0]) ) )
In [71]:
from nltk.corpus import stopwords
In [72]:
stopw = stopwords.words("english")
stopw.append("us")
In [73]:
def isStopword(word):
if word in stopw:
return(1)
return(0)
for x in model:
print( "\t".join( (str(x[1]), str(x[2]), x[0], str(isStopword(x[0]))) ) )
In [88]:
from nltk import pos_tag
In [95]:
tokens1 = word_tokenize(text1).lower())
posTokens = pos_tag(tokens1)
In [96]:
tags = list( set( [ x[1][0] for x in posTokens ] ) )
print(tags)
In [97]:
from collections import defaultdict
leftOfToken = defaultdict(Counter)
rightOfToken = defaultdict(Counter)
for i in range(len(posTokens)):
tag = posTokens[i][1][0]
token = (posTokens[i][0], tag)
if i > 0:
ltag = posTokens[i - 1][1][0]
leftOfToken[token][ltag] += 1
if i < len(posTokens) - 1:
rtag = posTokens[i + 1][1][0]
rightOfToken[token][rtag] += 1
In [98]:
for token in leftOfToken.keys():
leftVector = []
rightVector = []
for tag in tags:
leftVector.append(leftOfToken[token][tag])
rightVector.append(rightOfToken[token][tag])
print(" ".join([ str(x) for x in leftVector ]), " ".join([ str(x) for x in rightVector ]), token[0], token[1])
In [99]:
text2 = """A flight out of Austin, Texas, was delayed after
a pilot behaved in a way that caused passengers to believe
she was mentally unstable, a United Airlines spokesman said Sunday.
The pilot, whom CNN is not naming, boarded the plane in street
clothes and began speaking to passengers over the intercom,
spokesman Charlie Hobart said.
Passengers on Saturday's San Francisco-bound flight took to
social media to express concerns after the pilot spoke to
them about her divorce and the presidential election, among
other issues. """
tokens2 = word_tokenize(text2).lower())
posTokens2 = pos_tag(tokens2)
leftOfToken2 = defaultdict(Counter)
rightOfToken2 = defaultdict(Counter)
for i in range(len(posTokens2)):
token = (posTokens2[i][0], posTokens2[i][1][0])
if i > 0:
ltag = posTokens2[i - 1][1][0]
leftOfToken2[token][ltag] += 1
if i < len(posTokens2) - 1:
rtag = posTokens2[i + 1][1][0]
rightOfToken2[token][rtag] += 1
for token in leftOfToken2.keys():
leftVector = []
rightVector = []
for tag in tags:
leftVector.append(leftOfToken2[token][tag])
rightVector.append(rightOfToken2[token][tag])
print(" ".join([ str(x) for x in leftVector ]), " ".join([ str(x) for x in rightVector ]), token[0], token[1])
In [ ]: